{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stress Test性能测试\n",
    "- 千帆 Python SDK 提供了基于locust工具的对大模型服务进行快速压测以及性能评估的功能。该功能入口在Dataset对象的stress_test方法中。\n",
    "\n",
    "- 本文提供了一个使用multi_stress_test进行压测的示例，通过本文，您可以快速上手我们的多轮压测工具。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 由于Locust中使用了gevent库来保证高并发性能，而gevent的高并发依赖于monkey patching的非阻塞I/O机制，但该机制在Notebook环境中默认未开启。因此，在开始测试前，需要进行monkey patching操作。这样做是为了确保整个环境，包括IPython/Jupyter自己的组件，都使用gevent兼容的非阻塞版本，从而避免因混合使用阻塞和非阻塞操作导致的不一致性和潜在的死锁问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gevent import monkey\n",
    "monkey.patch_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "开始压测："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO][2024-07-18 11:37:39.841] dataset.py:430 [t:4370825664]: no data source was provided, construct\n",
      "[INFO][2024-07-18 11:37:39.863] dataset.py:276 [t:4370825664]: construct a file data source from path: ./data_file/stress_test.jsonl, with args: {}\n",
      "[INFO][2024-07-18 11:37:39.875] file.py:291 [t:4370825664]: use format type FormatType.Jsonl\n",
      "[INFO][2024-07-18 11:37:39.890] utils.py:359 [t:4370825664]: start to get memory_map from /Users/jianruitian/.qianfan_cache/dataset/Users/jianruitian/Documents/GitHub/bce-qianfan-sdk/cookbook/dataset/data_file/stress_test.arrow\n",
      "[INFO][2024-07-18 11:37:39.901] utils.py:284 [t:4370825664]: has got a memory-mapped table\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "[2024-07-18 11:37:46,744] \u001b[0;33m[NOTICE] Distributor[distributor] StopIteration occurs 2 times, and 2 users have been spawned.\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          2     0(0.00%) |    881     881     881    881 |    0.48        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         2     0(0.00%) |    881     881     881    881 |    0.48        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                             880    880    880    880    880    880    880    880    880    880    880      2\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                            880    880    880    880    880    880    880    880    880    880    880      2\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          2     0(0.00%) |     72      72      72     72 |    0.48        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         2     0(0.00%) |     72      72      72     72 |    0.48        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                              72     72     72     72     72     72     72     72     72     72     72      2\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                             72     72     72     72     72     72     72     72     72     72     72      2\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          2     0(0.00%) |     31      31      32     31 |    0.48        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         2     0(0.00%) |     31      31      32     31 |    0.48        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                              32     32     32     32     32     32     32     32     32     32     32      2\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                             32     32     32     32     32     32     32     32     32     32     32      2\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          3     0(0.00%) |   3263    2592    4069   3100 |    0.52        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         3     0(0.00%) |   3263    2592    4069   3100 |    0.52        0.00\n",
      "\n",
      "Response time percentiles (approximated)\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                            3100   3100   4100   4100   4100   4100   4100   4100   4100   4100   4100      3\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                           3100   3100   4100   4100   4100   4100   4100   4100   4100   4100   4100      3\n",
      "\n",
      "[2024-07-18 11:37:48,561] \u001b[0;32m[INFO] run completed.\u001b[0m\n",
      "[2024-07-18 11:37:48,590] \u001b[0;32m[INFO] Load Test Statistics\n",
      "user_num: 2\n",
      "worker_num: 2\n",
      "spawn_rate: 2\n",
      "model_type: ChatCompletion\n",
      "hyperparameters: None\n",
      "QPS: 0.52\n",
      "Latency Avg: 3.26\n",
      "Latency Min: 2.59\n",
      "Latency Max: 4.07\n",
      "Latency 50%: 3.1\n",
      "Latency 80%: 4.1\n",
      "FirstTokenLatency Avg: 0.75\n",
      "FirstTokenLatency Min: 0.47\n",
      "FirstTokenLatency Max: 0.88\n",
      "FirstTokenLatency 50%: 0.88\n",
      "FirstTokenLatency 80%: 0.88\n",
      "InputTokens Avg: 72.0\n",
      "OutputTokens Avg: 32.67\n",
      "TotalInputTokens Avg: 216.0\n",
      "TotalOutputTokens Avg: 98.0\n",
      "TotalQuery: 3\n",
      "SuccessQuery: 3\n",
      "FailureQuery: 0\n",
      "TotalTime: 8.68\n",
      "SuccessRate: 100.0%\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "[2024-07-18 11:37:53,638] \u001b[0;33m[NOTICE] Distributor[distributor] StopIteration occurs 4 times, and 4 users have been spawned.\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          3     0(0.00%) |   2817    2741    2931   2800 |    0.00        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         3     0(0.00%) |   2817    2741    2931   2800 |    0.00        0.00\n",
      "\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          3     0(0.00%) |   2817    2741    2931   2800 |    0.75        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         3     0(0.00%) |   2817    2741    2931   2800 |    0.75        0.00\n",
      "\n",
      "Response time percentiles (approximated)\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                            2800   2800   2900   2900   2900   2900   2900   2900   2900   2900   2900      3\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                           2800   2800   2900   2900   2900   2900   2900   2900   2900   2900   2900      3\n",
      "\n",
      "[2024-07-18 11:37:55,244] \u001b[0;32m[INFO] run completed.\u001b[0m\n",
      "[2024-07-18 11:37:55,264] \u001b[0;32m[INFO] Load Test Statistics\n",
      "user_num: 4\n",
      "worker_num: 2\n",
      "spawn_rate: 2\n",
      "model_type: ChatCompletion\n",
      "hyperparameters: None\n",
      "QPS: 0.75\n",
      "Latency Avg: 2.82\n",
      "Latency Min: 2.74\n",
      "Latency Max: 2.93\n",
      "Latency 50%: 2.8\n",
      "Latency 80%: 2.9\n",
      "FirstTokenLatency Avg: 0.72\n",
      "FirstTokenLatency Min: 0.65\n",
      "FirstTokenLatency Max: 0.83\n",
      "FirstTokenLatency 50%: 0.69\n",
      "FirstTokenLatency 80%: 0.83\n",
      "InputTokens Avg: 72.0\n",
      "OutputTokens Avg: 30.33\n",
      "TotalInputTokens Avg: 216.0\n",
      "TotalOutputTokens Avg: 91.0\n",
      "TotalQuery: 3\n",
      "SuccessQuery: 3\n",
      "FailureQuery: 0\n",
      "TotalTime: 6.66\n",
      "SuccessRate: 100.0%\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          1     0(0.00%) |   2389    2389    2389   2389 |    0.00        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         1     0(0.00%) |   2389    2389    2389   2389 |    0.00        0.00\n",
      "\n",
      "[2024-07-18 11:38:00,056] \u001b[0;33m[NOTICE] Distributor[distributor] StopIteration occurs 6 times, and 6 users have been spawned.\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          1     0(0.00%) |    754     754     754    754 |    0.41        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         1     0(0.00%) |    754     754     754    754 |    0.41        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                             750    750    750    750    750    750    750    750    750    750    750      1\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                            750    750    750    750    750    750    750    750    750    750    750      1\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          1     0(0.00%) |     72      72      72     72 |    0.41        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         1     0(0.00%) |     72      72      72     72 |    0.41        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                              72     72     72     72     72     72     72     72     72     72     72      1\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                             72     72     72     72     72     72     72     72     72     72     72      1\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          1     0(0.00%) |     30      30      30     30 |    0.41        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         1     0(0.00%) |     30      30      30     30 |    0.41        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                              30     30     30     30     30     30     30     30     30     30     30      1\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                             30     30     30     30     30     30     30     30     30     30     30      1\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          3     0(0.00%) |   2680    2389    2975   2700 |    0.80        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         3     0(0.00%) |   2680    2389    2975   2700 |    0.80        0.00\n",
      "\n",
      "Response time percentiles (approximated)\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                            2700   2700   3000   3000   3000   3000   3000   3000   3000   3000   3000      3\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                           2700   2700   3000   3000   3000   3000   3000   3000   3000   3000   3000      3\n",
      "\n",
      "[2024-07-18 11:38:01,910] \u001b[0;32m[INFO] run completed.\u001b[0m\n",
      "[2024-07-18 11:38:01,943] \u001b[0;32m[INFO] Load Test Statistics\n",
      "user_num: 6\n",
      "worker_num: 2\n",
      "spawn_rate: 2\n",
      "model_type: ChatCompletion\n",
      "hyperparameters: None\n",
      "QPS: 0.8\n",
      "Latency Avg: 2.68\n",
      "Latency Min: 2.39\n",
      "Latency Max: 2.98\n",
      "Latency 50%: 2.7\n",
      "Latency 80%: 3.0\n",
      "FirstTokenLatency Avg: 0.76\n",
      "FirstTokenLatency Min: 0.75\n",
      "FirstTokenLatency Max: 0.78\n",
      "FirstTokenLatency 50%: 0.76\n",
      "FirstTokenLatency 80%: 0.78\n",
      "InputTokens Avg: 72.0\n",
      "OutputTokens Avg: 33.0\n",
      "TotalInputTokens Avg: 216.0\n",
      "TotalOutputTokens Avg: 99.0\n",
      "TotalQuery: 3\n",
      "SuccessQuery: 3\n",
      "FailureQuery: 0\n",
      "TotalTime: 6.67\n",
      "SuccessRate: 100.0%\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "[2024-07-18 11:38:06,655] \u001b[0;33m[NOTICE] Distributor[distributor] StopIteration occurs 8 times, and 8 users have been spawned.\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          3     0(0.00%) |   2772    2584    3013   2700 |    0.82        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         3     0(0.00%) |   2772    2584    3013   2700 |    0.82        0.00\n",
      "\n",
      "Response time percentiles (approximated)\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                            2700   2700   3000   3000   3000   3000   3000   3000   3000   3000   3000      3\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                           2700   2700   3000   3000   3000   3000   3000   3000   3000   3000   3000      3\n",
      "\n",
      "[2024-07-18 11:38:08,591] \u001b[0;32m[INFO] run completed.\u001b[0m\n",
      "[2024-07-18 11:38:08,611] \u001b[0;32m[INFO] Load Test Statistics\n",
      "user_num: 8\n",
      "worker_num: 2\n",
      "spawn_rate: 2\n",
      "model_type: ChatCompletion\n",
      "hyperparameters: None\n",
      "QPS: 0.82\n",
      "Latency Avg: 2.77\n",
      "Latency Min: 2.58\n",
      "Latency Max: 3.01\n",
      "Latency 50%: 2.7\n",
      "Latency 80%: 3.0\n",
      "FirstTokenLatency Avg: 0.73\n",
      "FirstTokenLatency Min: 0.58\n",
      "FirstTokenLatency Max: 1.04\n",
      "FirstTokenLatency 50%: 0.58\n",
      "FirstTokenLatency 80%: 1.0\n",
      "InputTokens Avg: 72.0\n",
      "OutputTokens Avg: 30.67\n",
      "TotalInputTokens Avg: 216.0\n",
      "TotalOutputTokens Avg: 92.0\n",
      "TotalQuery: 3\n",
      "SuccessQuery: 3\n",
      "FailureQuery: 0\n",
      "TotalTime: 6.65\n",
      "SuccessRate: 100.0%\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "\u001b[0;36m首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36m输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\n",
      "[2024-07-18 11:38:14,874] \u001b[0;33m[NOTICE] Distributor[distributor] StopIteration occurs 10 times, and 10 users have been spawned.\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 首token延迟时间统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输入token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [stats]\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00\n",
      "\u001b[0m\n",
      "\u001b[0;36mSummary: 输出token数统计 [percentiles (approximated)]\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "\u001b[0m\n",
      "Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "POST     ERNIE-Bot                                                                          3     0(0.00%) |   4100    3827    4339   4100 |    0.58        0.00\n",
      "--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------\n",
      "         Aggregated                                                                         3     0(0.00%) |   4100    3827    4339   4100 |    0.58        0.00\n",
      "\n",
      "Response time percentiles (approximated)\n",
      "Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "POST     ERNIE-Bot                                                                            4100   4100   4300   4300   4300   4300   4300   4300   4300   4300   4300      3\n",
      "--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------\n",
      "         Aggregated                                                                           4100   4100   4300   4300   4300   4300   4300   4300   4300   4300   4300      3\n",
      "\n",
      "[2024-07-18 11:38:17,272] \u001b[0;32m[INFO] run completed.\u001b[0m\n",
      "[2024-07-18 11:38:17,305] \u001b[0;32m[INFO] Load Test Statistics\n",
      "user_num: 10\n",
      "worker_num: 2\n",
      "spawn_rate: 2\n",
      "model_type: ChatCompletion\n",
      "hyperparameters: None\n",
      "QPS: 0.58\n",
      "Latency Avg: 4.1\n",
      "Latency Min: 3.83\n",
      "Latency Max: 4.34\n",
      "Latency 50%: 4.1\n",
      "Latency 80%: 4.3\n",
      "FirstTokenLatency Avg: 0.88\n",
      "FirstTokenLatency Min: 0.64\n",
      "FirstTokenLatency Max: 1.02\n",
      "FirstTokenLatency 50%: 1.0\n",
      "FirstTokenLatency 80%: 1.0\n",
      "InputTokens Avg: 72.0\n",
      "OutputTokens Avg: 30.67\n",
      "TotalInputTokens Avg: 216.0\n",
      "TotalOutputTokens Avg: 92.0\n",
      "TotalQuery: 3\n",
      "SuccessQuery: 3\n",
      "FailureQuery: 0\n",
      "TotalTime: 8.68\n",
      "SuccessRate: 100.0%\u001b[0m\n",
      "[2024-07-18 11:38:17,319] \u001b[0;32m[INFO] Log path: ['record/20240718T113739Z/run.log', 'record/20240718T113739Z/run.log', 'record/20240718T113739Z/run.log', 'record/20240718T113739Z/run.log', 'record/20240718T113739Z/run.log']\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "# 开启压测服务\n",
    "os.environ['QIANFAN_ENABLE_STRESS_TEST'] = \"true\"\n",
    "\n",
    "from qianfan.dataset import Dataset\n",
    "\n",
    "# 请在此处填写您的ak、sk\n",
    "os.environ[\"QIANFAN_ACCESS_KEY\"] = \"...\"\n",
    "os.environ[\"QIANFAN_SECRET_KEY\"] = \"...\"\n",
    "\n",
    "# 需要初始化一个数据集对象\n",
    "ds = Dataset.load(data_file=\"./data_file/stress_test.jsonl\")\n",
    "\n",
    "ds.multi_stress_test(\n",
    "    origin_users=2,# 初始并发\n",
    "    # workers=2,# 实际工作的进程数\n",
    "    # spawn_rate=2,# 生成user的速度\n",
    "    # model=\"ERNIE-Speed-8K\",\n",
    "    # model_type=\"ChatCompletion\", \n",
    "    endpoint=\"YourEndpoint\",# 压测服务的endpoint\n",
    "    rounds = 5,# 压测轮次\n",
    "    interval=2,# 轮次间加压数\n",
    "    first_latency_threshold=5,# 单位秒\n",
    "    # round_latency_threshold=20,# 单位秒\n",
    "    # success_rate_threshold=99.5,# 成功率阈值，百分制\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与stress_test不同的是，multi_stress_test的输出有更多的轮次，每轮发压都会展示一次压测结果。\n",
    "\n",
    "multi_stress_test的前端报告也与stresstest的前端报告不同，我们添加了对各个压测轮次的压测数据的展示以及对压测模型信息的展示。前端报告的样例见cookbook/dataset/data_file/performance_table.html。\n",
    "\n",
    "以下是各参数的说明："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **workers (int)**：指定发压使用的worker数目，每个worker为1个进程，默认为1个进程；\n",
    "- **origin_users (int)**：指定发压使用的初始user数，必须大于worker数目；每个worker负责模拟${users}/${workers}个虚拟用户；\n",
    "- **runtime (str)**：指定发压任务的最大运行时间，格式为带时间单位的字符串，例如（300s, 20m, 3h, 1h30m）；压测任务启动后会一直运行到数据集内所有数据都请求完毕，或到达该参数指定的最大运行时间；该参数默认值为'0s'，表示不设最大运行时间；\n",
    "- **spawn_rate (int)**：指定每秒启动的user数目；\n",
    "- **model (str)**：指定需要压测服务的模型名称。该参数与endpoint只能指定一个；\n",
    "- **endpoint (str)**：指定需要压测服务的url路径。该参数与model只能指定一个；\n",
    "- **model_type (str)**：指定被测服务的模型类型。 目前只支持'ChatCompletion'与'Completion两类'；默认值为'ChatCompletion'；\n",
    "- **hyperparameters (Optional[Dict[str, Any]])**：指定压测时使用的超参数；\n",
    "- **rounds (int)**：指定压测轮数；\n",
    "- **interval (int)**：指定压测轮之间的加压并发数，比如interval=2，则在第1轮压测结束后，会在第2轮开始时，额外启动两个user的并发，以此类推；\n",
    "- **first_latency_threshold (float)**：指定首句时延的阈值，超过该阈值会停止在本轮压测，单位为秒；\n",
    "- **round_latency_threshold (float)**：指定全长时延的阈值，超过该阈值会停止在本轮压测，单位为秒；\n",
    "- **success_rate_threshold (float)**：指定请求成功率的阈值，低于该阈值会停止在本轮压测，单位为百分比；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各项指标含义如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **user_num**: 压测使用的本轮user数，即本轮发压的并发数；\n",
    "- **worker_num**: 压测使用的worker数目，即进程数；\n",
    "- **spawn_rate**: 每秒启动的user数目；\n",
    "- **model_type**: 被压测服务的模型类型；\n",
    "- **hyperparameters**: 压测使用的超参数；\n",
    "- **QPS**：服务每秒实际处理的请求数；\n",
    "- **Latency Avg/Min/Max/50%/80%**：全长时延的平均值/最小值/最大值/50分位值/80分位值；\n",
    "- **FirstTokenLatency Avg/Min/Max/50%/80%**：首句时延的平均值/最小值/最大值/50分位值/80分位值；\n",
    "- **InputTokens Avg**：单次请求输入的token长度平均值；\n",
    "- **OutputTokens Avg**：单次请求输出的token长度平均值；\n",
    "- **TotalQuery/SuccessQuery/FailureQuery**：总请求数/成功请求数/失败请求数；\n",
    "- **TotalTime**：总运行时间；\n",
    "- **SuccessRate**：请求成功率；"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "42453736f0d03c53e6ad75478ebb5fcf2abe6d38094996bfa938cecc75099885"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
